DEVISE A framework for the evaluation of Internet search engines

This project investigates the feasibility of the use of user-satisfaction as a multidimensional evaluative construct of search engines. Search engine developments are reviewed to reveal a range of indexing and retrieval techniques that may assist casual users in the information retrieval task. Yet few evaluation studies have considered the impact of system features, in particular those with which the user interacts for search assistance. A broad review of retrieval system evaluation highlights the complex environment in which measures of both the utility of the search results and the usability of the system are sought from a userperspective. Our proposed approach for a user-centered evaluation is based on a conceptual framework in which user-satisfaction is characterised as a variable dependent on system features and functions and expressed in a moderating context of user-task requirement. Towards this end, the research reported here focuses on the definition of the construct of user satisfaction on the multi dimensions of the retrieval process, an expression of what a typical user is trying to do. Empirical work was then undertaken to test the feasibility and potential value of the implementation of the framework for the evaluation of three search engines. Initial results are presented which provide a degree of understanding of how users are satisfied and on what criteria. This provides the basis on which we make recommendations for the refinement of the multidimensional framework and its use as a methodology for the evaluation of search engines from a user perspective. For further information or correspondence please contact the Project Director or the Project Research Fellow at the address below: Dr Frances Johnson Jillian Griffiths Project Director Research Fellow Centre for Research in Library & Information Management (CERLIM) Department of Information & Communications The Manchester Metropolitan University, Geoffrey Manton Building Manchester, M15 6LL Tel: +161-247-6156; Fax: +161-247-6351 [email protected] , [email protected]  Resource: The Council for Museums, Archives and Libraries 2001 The opinions expressed in this report are those of the authors and not necessarily those of Resource: The Council for Museums, Archives and Libraries Library and Information Commission Research Report 100